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ABSTRACT 

Introduction: Accreditation programmes aim to 
improve the quality and safety of health services, and 
have been widely implemented. However, there is 
conflicting evidence regarding the outcomes of existing 
programmes. The Accreditation Collaborative for the 
Conduct of Research, Evaluation and Designated 
Investigations through Teamwork-Current Accreditation 
Processes (ACCREDIT-CAP) project is designed to 
address key gaps in the literature by evaluating the 
current processes of three accreditation programmes 
used across Australian acute, primary and aged care 
services. 

Methods and design: The project comprises three 
mixed-method studies involving documentary analyses, 
surveys, focus groups and individual interviews. Study 
samples will comprise stakeholders from across the 
Australian healthcare system: accreditation agencies; 
federal and state government departments; consumer 
advocates; professional colleges and associations; and 
staff of acute, primary and aged care services. Sample 
sizes have been determined to ensure results allow 
robust conclusions. Qualitative information will be 
thematically analysed, supported by the use of textual 
grouping software. Quantitative data will be subjected to 
a variety of analytical procedures, including descriptive 
and comparative statistics. The results are designed to 
inform health system policy and planning decisions in 
Australia and internationally. 
Ethics and dissemination: The project has been 
approved by the University of New South Wales Human 
Research Ethics Committee (approval number HREC 
10274). Results will be reported to partner organisations, 
healthcare consumers and other stakeholders via peer- 
reviewed publications, conference and seminar 
presentations, and a publicly accessible website. 
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INTRODUCTION 

Health service accreditation 

Health service accreditation programmes are 
complex system-level interventions that aim to 
improve the quality and safety of patient care 



Article focus 

■ This project involves three studies that aim to 
evaluate current health service accreditation pro- 
cesses in three domains: acute, primary and 
aged care. Specific issues to be examined 
include: the strengths and consequences of dif- 
ferent accreditation models; the critical elements 
of accreditation processes that stimulate behav- 
ioural change in health services and healthcare 
professionals; and the development and use of 
accreditation standards. 

Key messages 

■ Accreditation programmes are implemented 
worldwide to drive and transfer quality and 
safety improvements across a variety of health 
service organisations. 

■ Evidence supporting the capacity of accreditation 
programmes to promote high-quality and safe 
organisational and clinical performance is both 
limited and contested. 

■ This project will involve multimethod approaches 
to help build the evidence base regarding CAP. 

Strengths and limitations of this project 

■ The strength of this project is its examination of 
accreditation programmes across acute, primary 
and aged care domains, and comprehensive mul- 
timethod approach designed to produce results of 
practical utility across the health system. 

■ A limitation is the project's focus on accredit- 
ation programmes from one country. The results 
may require additional interpretation to be trans- 
ferable to other countries. 



by monitoring the performance of health ser- 
vices against predetermined standards. 1 2 
Assessments are usually made by external 
accrediting or certifying agencies using stan- 
dards developed in consultation with health- 
care industry experts. 3 The participation of 
health services in accreditation programmes 
can be voluntary legislatively mandated or 
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encouraged through the use of financial incentives 
offered by governments or insurance agencies. 4 While 
originally designed for use within acute care settings, 
accreditation programmes are now available for a range 
of organisations, including primary and aged care ser- 
vices.'' Accreditation cycles commonly last between 3 and 
5 years and involve all, or a mixture of the following activ- 
ities: organisational self-assessment; on-site assessment by 
accreditation surveyors; the provision of a summative 
report to organisations that may include improvement 
recommendations and subsequent actions implemented 
by health services to address recommendations. 1 2 



Health service accreditation evidence base 

Given the large number of countries with accreditation 
systems, there is a large yet undetermined global invest- 
ment in health service accreditation. 6 7 Despite this 
expenditure, there is limited research that has demon- 
strated the effectiveness of programmes in achieving their 
stated aims. 8 Recent literature reviews have found accredit- 
ation may promote changes to organisational processes 
associated with quality of care. 4 5 8-11 However, several 
factors reduce the generalisability of these findings: the 
small number of empirical studies; limited use of experi- 
mental study designs; difficulty of aggregating results from 
individual studies due to their varied research contexts, 
that is, different country, healthcare domain and accredit- 
ation programme settings; and the infrequent use of 
patient outcome indicators for evaluation. 4 5 8-11 
Furthermore, there is limited understanding of the spe- 
cific processes within accreditation programmes that are 
most capable of promoting quality and safety improve- 
ments within health service organisations. 5 These issues 
highlight the pressing need for additional rigorous 
accreditation research, which has prompted calls to 
strengthen the evidence base. 12 13 This project aims to link 
researchers, policy bodies and industry partners to provide 
a response to the global need for further, high-quality 
health service accreditation research. 



Project overview 

The project is one component of the ACCREDIT 
(Accreditation Collaborative for the Conduct of Research, 
Evaluation and Designated Investigations through 
Teamwork) research collaboration. 14 ACCREDIT involves 
research and industry partners: researchers in the Centre 
for Clinical Governance Research and Centre for Health 
Systems and Safety Research in the Australian Institute of 
Health Innovation at the University of New South Wales 
(UNSW); three major Australian healthcare accreditation 
agencies (Australian Council on Healthcare Standards 
(ACHS), Australian General Practice Accreditation 
Limited (AGPAL) and Aged Care and Standards 
Accreditation Agency (ACSAA) ) ; leading quality improve- 
ment policy bodies (the Australian Commission on Safety 
and Quality in Health Care and the Clinical Excellence 
Commission) ; and an international advisory group consist- 
ing of prominent quality and safety researchers from 
Spain, Sweden and the UK The research is funded by the 
Australian Research Council through its industry Linkage 
Programme, 15 which aims to promote collaboration 
between researchers and industry groups to generate rigor- 
ous research with practical implications for Australian 
society. The collaboration has successful experience in this 
form of research. 16 This research protocol details an evalu- 
ation of current processes employed by Australian health 
service accreditation programmes within the ACCREDIT 
collaboration; known as the Current Accreditation 
Processes, ACCREDIT-CAP protocol. 



METHODS AND ANALYSIS 

Three studies will be undertaken to evaluate CAP across 
acute, primary and aged care domains (figure 1). Each 
study has specific questions and uses multiple methods 
to address them. 

As illustrated in table 1, a range of research techniques 
will be applied, including documentary analyses, focus 
groups, interviews and surveys. The use of different 
research methods will address the potential limitations 



Figure 1 Research studies and 
questions. 



Accreditation models 
a) What arc the relative 
strengths and 
consequences of different 
accreditation models 9 




Standards and thcii impact 

a) How arc standards developed 
and used 9 

b) How do standards incorporate 
evidence, and influence the 
expertise of clinicians, 
managers, and policymakers 9 

:) How does the application of 
standards promote change in 
organisational performance and 
clinical practice 9 



Critical elements of 

accreditation 
What are the critical 
elements of the 
accreditation process that 
stimulate improvement? 
What drives behaviour 
change in health service 
organisations and their 
staff 9 
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Table 1 Study research questions, approaches and methods 



Study 



Research questions 



Summary of research 
approaches, tasks, and scope 



Summary of methods, sample 
sizes, data requirements, 
analysis and design features 



Accreditation 
models 



Critical elements 
of accreditation 



Standards and 
their impact 



What are the relative strengths 
and consequences of different 
accreditation models? 



What are the critical elements 
of the accreditation process 
that stimulate improvement? 
What drives behaviour change 
in health service organisations 
and their staff? 



How are standards developed 
and used? 

How do standards incorporate 
evidence, and influence the 
expertise of clinicians, 
managers and policymakers? 
How does the application of 
standards promote change in 
organisational performance 
and clinical practice? 



Undertake a multimethod 
evaluation of three accreditation 
programmes: those of ACHS, 
AG PAL and ACSAA 



Assess programme elements (eg, 
self-assessment, clinical indicators, 
surveyor visits and accreditation 
reports) and describe their role in 
promoting improvement 



Examine the development of 
standards and their application 
across different accreditation 
programs, selecting and 
investigating a sample of standards 
to determine their sources (eg, 
public inquiries, adverse events, 
international guidelines) and how 
they were developed and applied 



Interview key stakeholders of 
three accreditation agencies 
(n=18) 

Conduct a web-based survey of 
staff from accredited acute, 
primary and aged care services 
(n=~300) 



Run focus groups of 
stakeholders drawn from 
accreditation agencies (n=6 
focus groups) and jurisdictional 
health departments (n=8 focus 
groups) and 15 focus groups 
involving staff from accredited 
acute (n=5), primary (n=5) and 
aged care services (n=5) 
Conduct a web-based survey of 
staff from accredited acute, 
primary and aged care services 
(n=~300) 

Gather examples of how the 
respective accreditation 
elements drive change 



Use documentary analysis and 
focus groups (n=3) to 
retrospectively analyse the 
development of standards, 
assessing the use of evidence 
and engagement of stakeholders 
Conduct a survey of staff from 
accredited organisations, 
investigating how standards are 
applied and how they promote 
change (n=~300). From these 
data, identify for detailed 
analysis case study sites in 
which standards have promoted 
measurable change 
Conduct case studies (n=5) of 
specified key standards 
(evaluation of care, documented 
policies, the quality improvement 
system, health records and 
infection control) to identify 
factors related to organisational 
change. Employ focus groups 
(n=15) and obtain organisational 
data to measure the extent of this 
change 
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associated with any single study tool or measure. The 

1 7 

collation of varied data sources, that is, triangulation, 
will strengthen the credibility and generalisability of 
results by facilitating comparative confirmation of 
findings. 

Study 1: accreditation models 

Study 1 aims to examine the relative strengths and con- 
sequences of the main accreditation programmes of 
ACHS, AGPAL and ACSAA. 

Interviews with accreditation agency stakeholders 

Six one hour interviews will be held with stakeholders of 
each partner accreditation agency (n=18 interviews in 
total). Each agency will provide the contact details of 
representatives from organisations they consider key 
stakeholders. The following groups are likely to be nomi- 
nated: federal and state government agencies; health- 
care professional colleges and associations; accreditation 
agency councils, boards, staff and surveyors; and con- 
sumer representatives. A sample of stakeholders will be 
purposively selected from the lists provided by each 
agency in order to include representatives from a variety 
of groups. Based on the research team's prior accredit- 
ation research experience, 18 19 it is probable that 
thematic saturation 20 (ie, when no new themes or cat- 
egories emerge from the data) will be reached within 
the proposed sample size. 

Specific issues to be examined within interviews will 
be finalised through collaboration between researchers 
and industry partners. Based on previous accreditation 
literature reviews 5 and studies 21 22 undertaken by the 
research team, the interview schedule is likely to include 
a combination of the following topics: philosophical 
positions underlying accreditation programmes (eg, 
regulatory compliance or continuous quality improve- 
ment) ; effects of different accreditation agency manage- 
ment structures and administrative processes; the roles 
and influence of key stakeholders; intended and unin- 
tended impacts of accreditation programmes on health 
services; incentives encouraging health service participa- 
tion in accreditation programmes; standards develop- 
ment processes; the roles of surveyors; issues 
surrounding the public disclosure of accreditation 
results and the role of short notice and patient journey 
survey methodologies. 

Recruitment will occur via emails, in which the study 
and participant roles within it will be explained using 
information and consent forms approved by the UNSW 
Human Research Ethics Committee. 23 If responses are 
not received within 2 weeks, the research team will 
follow up nominated stakeholders via telephone to 
request their participation. Interview locations will be 
determined through discussion with potential partici- 
pants, and are likely to occur across several Australian 
cities, states and territories. 

Where possible, face-to-face rather than telephone 
interviews will be arranged to increase participant 



comfort in speaking openly regarding potentially sensi- 
tive issues. Other advantages of the face-to-face approach 
include increased respondent attentiveness to questions 
and the opportunity for interviewers to detect non-verbal 
feedback. 24 Two experienced research team members 
(RH and DG) will conduct interviews. Training will be 
organised prior to data collection to ensure a uniform 
approach, which will produce higher reliability and more 
internally valid cross-comparisons of data. 25 

The semi-structured interview technique will be 
employed to balance uniformity against variation of 
responses. Data resulting from each interview will be suf- 
ficiendy similar to allow stakeholder views to be com- 
pared. However, use of a non-prescriptive interviewing 
approach will allow respondents to discuss issues not 
explicitly raised within the interview schedule. New con- 
cepts can be included for subsequent interviews. The 
continual process of adapting the interview schedule will 
be critical, as the aim is not validation of pre-existing 
hypotheses, but identification of emergent issues in 
order to generate new understandings. Interviews will be 
digitally recorded and professionally transcribed. 

Thematic analysis of interview transcriptions will be 
undertaken inductively, supported by the use of the 
textual grouping software, NVivo V.9 (Doncaster, 
Australia) . 26 Such programmes are used to facilitate sys- 
tematic classification of the data. 27 Three general princi- 
ples will guide the analysis: include all major rival 
interpretations, address only the most significant aspects 
of the data and use prior knowledge and experience to 
inform analytical decisions. 20 Preliminary development 
of thematic categories (ie, key topics discussed by 
respondents) and initial coding will be conducted by 
the first author, who has extensive qualitative data ana- 
lysis experience. 28-31 After discussion among the 
research team to cross-check the validity and relevance 
of developed themes, categories will be refined and 
adjusted until a coherent scheme is developed that 
admits all instances and is applied to all interview data. 
To determine the reliability of categorisation by the first 
author, 50 randomly selected coded sentences will be 
assessed by a second researcher issued with category 
definitions. The level of inter-rater reliability will be 
assessed using Cohen's Kappa statistic, which measures 
the inter-rater agreement, accounting for that expected 
by chance. 32 A result less than 0.7 will necessitate 
further refinement of categories until a higher level of 
agreement is achieved. 

Survey of staff from accredited health services 

A minimum of 300 web-based surveys using KeySurvey 33 
software (WorldAPP, Braintree, MA) will be collected 
from a purposive sample of staff from accreditation 
agency member organisations, that is, acute, primary 
and aged care services enrolled in accreditation pro- 
grammes. Advantages of web-based surveys compared to 
paper-based surveys are cost reduction, speed and facili- 
tation of more complex questionnaire routing. 34 
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Respondents will be asked to evaluate 25 statements con- 
cerning key characteristics of accreditation programmes 
using a five-point Likert scale 35 ranging from 'strongly 
agree' to 'strongly disagree'. Survey questions will be 
drawn from the interview schedule and informed by ana- 
lysed interview data. Participant demographic informa- 
tion will also be collected, including: age, professional 
background, level of experience, highest qualification, 
occupational position and years in current position, 
organisation and industry. In total, the survey will take 
approximately 25 min to complete. 

A pilot study will be undertaken to ensure that instruc- 
tions and questions are comprehensible (eg, appropriate 
terminology and jargon are used), the length of time 
required for completion is feasible and software func- 
tions effectively. Senior management teams of partner 
agencies (n=15 individuals in total), and colleagues of 
the research team with healthcare professional experi- 
ence (n=10) will comprise the pilot sample and provide 
critical feedback. The survey design will be amended as 
necessary. 

Following the pilot study, industry partners will provide 
a catalogue of their member health services, including 
details regarding organisational locations (metropolitan, 
regional or remote), sizes (number of staff) and type of 
ownership (public or private). Lists will be stratified 
based on these characteristics, with at least 100 services 
from each healthcare domain purposively selected to 
ensure a range of organisations are included in the 
sample. Through prior collaboration between research- 
ers and industry partners, the sample size was determined 
to ensure the study is feasible, yet remains capable of pro- 
ducing industry and policy-relevant results. Individuals 
responsible for accreditation processes in each selected 
health service (eg, quality managers) will be invited to 
participate via emails from partner accreditation agen- 
cies. Information and consent forms will outline the 
study and potential participant's role within it. Follow-up 
of non-respondents via telephone will occur after 
1 month to maintain an adequate response rate (over 
65%). 36 A review of accreditation literature by the 
research team' identified considerable variation in the 
response rates of similar studies, which impedes rigorous 
estimation of likely rates that may be achieved for this 
and other surveys within the project. Nonetheless, the 
facilitation of recruitment by partner agencies and the 
targeting of individuals with a professional focus on 
accreditation should ensure a positive outcome. 

Several methods will be used to analyse the survey 
data, including: generalised linear modelling applied to 
Likert-type outcomes, such as proportional odds model- 
ling, 37 categorical data analysis techniques, 38 supervised 
and unsupervised statistical learning, 39 and factor ana- 
lysis. 40 Exact and efficient inferential procedures will be 
used to ensure statistical validity of obtained analytical 
outcomes and maximum utilisation of available sample 
sizes. 41 42 Empirical patterns will be examined by stratify- 
ing quantitative data with respect to key alternative 



factors based on respondent and organisational charac- 
teristics. Participant responses will be compared both 
within and between stakeholder groups and accredit- 
ation domains to identify variations regarding views of 
key themes. Potential confounding factors and causal 
links will be examined based on empirical evidence. 

Study 2: critical elements of accreditation 

Study 2 aims to deduce the critical elements of accredit- 
ation processes that stimulate improvement, and identify 
factors that drive behaviour change in health services 
and their staff. 

Focus groups with accreditation agency stakeholders, 
jurisdictional health department representatives and staff 
from accredited health services 

Focus groups will be conducted with accreditation 
agency stakeholders (n=6 focus groups), jurisdictional 
health department representatives (n=8 focus groups) 
and staff from accredited acute, primary and aged care 
services (n=15). Each group will involve between 5 and 
10 participants and last approximately 1 h. Focus groups 
were considered an ideal methodology for this study 
activity due to their ability to capture communication 
between research participants, which can help generate 
additional insights to one-on-one interviews. 43 

Accreditation agency stakeholders will be purposively 
drawn from the lists used to guide the interview sample 
selection for study one. Where possible, different groups 
will be invited to participate, but the same recruitment 
strategy will be employed. Staff from accredited health 
services will be recruited at educational workshops run by 
accreditation agencies in metropolitan and regional loca- 
tions throughout Australia. Focus groups regarding each 
accreditation programme will be arranged in different 
states to obtain a national perspective. Two experienced 
researchers (RH and DG) will conduct the focus groups. 

Specific accreditation elements to be explored within 
focus groups will be informed by research literature 5 44 
and the combined accreditation knowledge accumulated 
by research partners. Topics may include: accreditation 
agencies (eg, their credibility and transparency), 
accreditation standards (eg, development processes), 
performance measures/indicators/outcomes (eg, their 
validity and associated collection and collation difficul- 
ties), surveyors (eg, reliability and training/educational 
support issues) and levels of consumer involvement (eg, 
the role of consumer surveyors and issues surrounding 
public disclosure of accreditation results). Focus groups 
will be digitally recorded and transcribed. Transcriptions 
will be analysed using the same methods as for the inter- 
view data in study 1 . 

Survey of staff from accredited health services 

A web-based survey will be implemented to triangulate 
and supplement the focus group results. The survey will 
ascertain respondents' views of the relative importance 
of accreditation elements and their role in promoting 
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organisational improvement. Participants will be drawn 
from staff of accredited organisations across acute, 
primary and aged care settings (~n=300). Equivalent 
survey methods will be employed as for study 1, includ- 
ing: sampling frame and rationale (ie, a purposively 
selected sample of a sufficient size to produce 
industry-relevant results), questionnaire design (ie, 25 
Likert scale questions), recruitment strategy (email invi- 
tations from accreditation agencies) and analytical tech- 
niques. A free-text section will allow participants to 
describe examples where specific accreditation elements 
have promoted measurable change in their organisation. 

Study 3: standards and their impact 

Study 3 aims to examine: how standards are developed 
and used; how they incorporate evidence and influence 
the expertise of clinicians, managers and policymakers; 
and how their application promotes change in organisa- 
tional performance and clinical practice. 

Retrospective analysis of standards development processes 

The research team will retrospectively analyse the activ- 
ities and processes used to develop standards for the 
main accreditation programme of each partner agency. 
For data collection, systematic examination of accredit- 
ation agency standards development committee and 
workshop reports, that is, 'documentary analysis', 45 will 
be employed in combination with focus groups (n=3) 
involving between 5 and 10 individuals involved in stan- 
dards development processes for each agency. Focus 
groups will be conducted by two experienced research- 
ers (RH and DG) . The main topic will be factors pro- 
moting standards development. Based on prior work of 
the research team,'' key factors likely to be discussed 
include public inquiries, reporting of adverse events or 
International Clinical Guidelines. Focus groups will be 
recorded, transcribed and thematically analysed. Sample 
selection, recruitment and access to relevant reports will 
be facilitated by partner agencies. 

Survey of staff of accredited health services 

A web-based survey will investigate the views of staff from 
accredited health services regarding the application of 
standards and how they promote change (~n=300 
respondents). Specific standards-related issues to be 
explored will be drawn from prior studies conducted by 
the research team 1 5 and may include: use of empirical 
evidence for development; clarity of organisational 
requirements; revision processes; impacts on administra- 
tive processes, clinical care and patient/ client satisfac- 
tion; roles of surveyors; and whether standards should 
be developed independently of assessment agencies. A 
free-text section will ask participants to describe exam- 
ples where standards have produced measurable 
change. Survey software and design, sample selection, 
recruitment and analysis methods will mirror those used 
for prior survey studies. 



Case studies of key accreditation programme standards 

Case studies (n=5) of specified key standards — evaluation 
of care, documented policies, the quality improvement 
system, health records and infection control — will be 
conducted. Through prior literature reviews, studies and 
collaboration with industry and policy partners, these 
standards were identified as critical influences on the 
utility of accreditation programmes. For each case study, 
factors related to organisational change resulting from 
standards will be examined within a single health service 
from acute, primary and aged care domains. The 
sample will be purposively selected using the free-text 
section of study 3 survey results. 

Organisational performance data will be drawn from 
accreditation and clinical indicator reports where avail- 
able. The attitudes and beliefs of health service staff will 
be obtained through focus groups (n=15). Between 5 
and 10 individuals from each organisation will be invited 
to participate in a Th focus group, in which participants 
will be questioned regarding the impacts of key stan- 
dards. The focus group schedule will be developed based 
on results from previous research activities for study 
three. Two experienced team members (RH and DG) 
will conduct the focus groups. The resulting data will be 
analysed using the same techniques as for other qualita- 
tive studies previously described in this protocol. 

ETHICS AND DISSEMINATION 

The UNSW Human Research Ethics Committee has 
approved the proposals for each study, as well as related 
information sheets and consent forms summarising the 
research and roles of potential participants. All project 
data will be de-identified by removing statements identi- 
fying participants prior to their use in published materi- 
als. All survey, interview, focus group and case study data 
will be stored securely in password-protected folders 
accessible only to the research team. In accordance with 
Australian National Health and Medical Research 
Council (NHMRC) guidelines, 46 all project data will be 
destroyed after a minimum of 7 years. Participants will 
be provided with contact details of the research team to 
enable them to address any potential queries or con- 
cerns. Complaints will be systematically recorded and 
actioned in accordance with UNSW guidelines. 

Participants will be informed of their right to access 
their own results, and the overall results of the research, 
in accordance with NHMRC guidelines. Journal articles 
developed from the research will be sent to all project 
partners for distribution to participants, and 
de-identified results will be published on a publicly 
accessible website. Conference and seminar presentations 
will be made to Australian and international healthcare 
stakeholders. 

CONCLUSION 

The project involves a collaborative partnership between 
major Australian healthcare accreditation agencies, quality 
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improvement bodies and researchers. The three multi- 
method, triangulated studies presented in this protocol 
will evaluate CAP. The research and professional knowl- 
edge, skills and experience harnessed by the research part- 
nership provides sufficient capacity to design and 
implement a project capable of producing wide-ranging 
industry and policy-relevant results. The project will 
address knowledge gaps concerning the utility and reliabil- 
ity of health service accreditation, and increase the trans- 
parency and credibility of accreditation processes. 
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